# Regarding file matching

## Support for Gitignore

### pre-analysis

based on CLI `man(ual)` git ignore : `man 5 gitignore`

```
GITIGNORE(5)                                                                                                           Git Manual                                                                                                             GITIGNORE(5)

NAME
       gitignore - Specifies intentionally untracked files to ignore
```

specifically:

```

DESCRIPTION
       [...]
       Each line in a gitignore file specifies a pattern.
       [...]

PATTERN FORMAT
       [...]
       See fnmatch(3) and the FNM_PATHNAME flag for a more detailed description.
```

the algorithm to support `.gitignore` syntaxes for filtering we need requires the following charateristics:

- multiple patterns match
- `fnmatch` patterns support

### Gathering the list of files

Ideally speaking we want to find a solution that would support the generation of the filepaths list based on the "gitignore" in one go.

The standard filesystem API from nodejs 20 is not able to support glob patterns as of today, it's merely able to generate a list of every single file-path recursively.

#### How does other do it?

##### Glob - how does it generate the file list?

based on my understanding, the creator of `Glob` has also developed a dedicated file-system optimization library he uses in the Glob package, which is [path-scurry](https://www.npmjs.com/package/path-scurry#:~:text=For%20resolving,resolution).

> For resolving string paths, PathScurry ranges from 5-50 times faster than path.resolve on repeated resolutions, but around 100 to 1000 times slower on the first resolution.
> [...]
> With default settings on a folder tree of 100,000 items, consisting of around a 10-to-1 ratio of normal files to directories, PathScurry performs comparably to @nodelib/fs.walk, which is the fastest and most reliable file system walker I could find.
> As far as I can tell, it's almost impossible to go much faster in a Node.js program, just based on how fast you can push syscalls out to the fs thread pool.

I understand that under the hood `Glob` is NOT applying the matching pattern at the "matching" step itself, it's instead generating a list of all the files and later on filtering it down with `regexp` string matching.

I also understand that the optimization path-scurry are providing to the file-list generation are not necessarly beneficial to our scenario and are making a trade-off of memory in favory of execution time, which in CI & CD environments may not be desiderable due to the reduced RAM of the environments, we usually prefeer to have the pipeline run 2x longer but requiring 1/2 the RAM resources rather then taking 1/2 the time but reserving 2x the RAM.

[from scurry docs](https://www.npmjs.com/package/path-scurry#:~:text=PathScurry%20makes,can)

> PathScurry makes heavy use of LRUCache to efficiently cache whatever it can, and Path objects remain in the graph for the lifetime of the walker

##### [Glob](https://www.npmjs.com/package/glob#:~:text=testing,characters) - gitignore support

based on my current understanding, the node package `glob` is not specifically about supporting all the possible syntaxes that are meant to be supported by the gitignore patterns:

> testing, fast-glob is around 10-20% faster than this module when walking over 200k files nested 4 directories deep 1.
> However, there are some inconsistencies with Bash matching behavior that this module does not suffer from

and it's instead trying to create a bash compliant list of files paths based on bash patterns.

for this reason to ensure 1-1 `Glob` matching would require still a further pattern matching on the generated list and I understand we would not necessarly benefit from the trades-off glob is making under the hood.

#### What I will do

##### step 1 - generate the file-list via native fs readDir

I will use the native node implementation to ensure maximum compatibility and to not encur the risk that the glob abstraction may incurr due to the heavy use of the LRUCache, relying on node's own performance optimization. I think that if any enhancement on the performance/memory is required it would be easier to manage if we don't deal with Glob out-of-the-box optimization and start from the bottom-line

##### step 2 - further filtering - gitignore compliant

~~I have found out [micromatch](https://github.com/micromatch/micromatch)~~

~~which seems to be used by `square`, `webpack`, `babel core`, `yarn`, `jest`, `taro`, `bulma`, `browser-sync`, `documentation.js`, `stylelint`, `nyc`, `ava`~~

~~and seems to be matching our requirements.~~

after live testing, micromatch is not following the gitignore specs,
see [github issue](https://github.com/micromatch/micromatch/issues/244) for reference

I will use [ignore](https://www.npmjs.com/package/ignore) instead, which
[seems to be doing exactly what we want](https://stackblitz.com/edit/node-61bbcz?file=ignore.js)